Skip to content

[Firebase AI] Add support for thought summaries #15096

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 30 commits into from
Aug 18, 2025
Merged

[Firebase AI] Add support for thought summaries #15096

merged 30 commits into from
Aug 18, 2025

Conversation

andrewheard
Copy link
Contributor

@andrewheard andrewheard commented Jul 8, 2025 •
edited
Loading

Added support for returning thought summaries when using thinking-compatible models.

Thought summaries may be enabled by setting includeThoughts: true in the ThinkingConfig. For example:

let model = FirebaseAI.firebaseAI().generativeModel(
  modelName: "gemini-2.5-flash",
  GenerationConfig(thinkingConfig: ThinkingConfig(includeThoughts: true))
)

@gemini-code-assist Gemini Code Assist
Copy link
Contributor

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

paulb777
paulb777 reviewed Jul 14, 2025
View reviewed changes
paulb777
paulb777 reviewed Jul 28, 2025
View reviewed changes
@gemini-code-assist Gemini Code Assist
Copy link
Contributor

Summary of Changes

This pull request introduces robust support for handling and exposing 'thought summaries' from Firebase AI models. It involves significant refactoring of the internal content representation to distinguish thought-related parts, updates to the public API for accessing these summaries, and modifications to content aggregation logic to ensure proper separation. Comprehensive tests have been added to verify the new functionality.

Highlights

  • Core Data Model Enhancement: I've refactored the internal representation of ModelContent parts by introducing a new InternalPart struct with isThought and thoughtSignature properties. This allows the SDK to differentiate between regular content and the model's internal 'thought' processes, and to associate a unique signature with certain thought-related parts.
  • Public API for Thought Summaries: I've added a new public property thoughtSummary: String? to GenerateContentResponse, providing a convenient way to access the aggregated text of thought parts from the model's response. The existing text property now specifically returns non-thought text.
  • ThinkingConfig Expansion: The ThinkingConfig struct now includes an includeThoughts: Bool? parameter in its initializer. This allows developers to explicitly request or suppress thought summaries from the model, offering more granular control over the response content.
  • Content Aggregation Logic Update: The aggregatedChunks function in Chat.swift has been updated to correctly separate and combine regular text parts from thought text parts, ensuring that thought summaries are not inadvertently merged with the main response text.
  • Filtering Thought Parts from Other Types: I've updated the GenerateContentResponse to filter out thought parts when accessing functionCalls and inlineDataParts, ensuring that these properties only return relevant content types and not internal thought representations.
  • Comprehensive Testing: I've added extensive unit and integration tests to validate the new thought summary functionality, covering both unary and streaming generateContent calls, and scenarios involving function calling, across Google AI and Vertex AI models.
Changelog
  • FirebaseAI/Sources/Chat.swift
    • Refactored aggregatedChunks to correctly separate and aggregate 'thought' text from regular text, preventing their unintended combination.
    • Updated the internal parts array to use the new InternalPart type, which carries isThought and thoughtSignature metadata.
  • FirebaseAI/Sources/GenerateContentResponse.swift
    • Added a new public property thoughtSummary: String? to access aggregated thought text.
    • Refactored the existing text property to use a new private helper text(isThought:) to retrieve only non-thought text.
    • Modified functionCalls and inlineDataParts properties to exclude parts marked as isThought.
  • FirebaseAI/Sources/ModelContent.swift
    • Moved InternalPart to a top-level struct and added isThought: Bool? and thoughtSignature: String? properties.
    • Introduced InternalPart.OneOfData enum to encapsulate different data types within InternalPart.
    • Updated ModelContent's parts computed property and initializers to correctly map between public Part types and the new InternalPart structure, preserving thought metadata.
    • Updated Codable conformance for InternalPart and InternalPart.OneOfData to handle serialization/deserialization of thought metadata.
  • FirebaseAI/Sources/Types/Internal/InternalPart.swift
    • Added isThought and thoughtSignature properties to ErrorPart for consistency, defaulting isThought to false.
  • FirebaseAI/Sources/Types/Public/Part.swift
    • Introduced isThought: Bool as a new requirement for the Part protocol.
    • Updated TextPart, InlineDataPart, FileDataPart, FunctionCallPart, and FunctionResponsePart to conform to the Part protocol's new isThought requirement, and added internal _isThought and thoughtSignature properties to propagate this information.
  • FirebaseAI/Sources/Types/Public/ThinkingConfig.swift
    • Added includeThoughts: Bool? property to ThinkingConfig to allow explicit control over thought summary inclusion in responses.
    • Updated the ThinkingConfig initializer to accept the new includeThoughts parameter.
  • FirebaseAI/Tests/TestApp/Tests/Integration/GenerateContentIntegrationTests.swift
    • Updated generateContentThinking test to use the new ThinkingConfig with includeThoughts and added assertions for thought parts.
    • Added a new integration test generateContentThinkingFunctionCalling to verify thought summaries and signatures in function calling scenarios.
  • FirebaseAI/Tests/Unit/GenerativeModelGoogleAITests.swift
    • Added unit tests testGenerateContent_success_thinking_thoughtSummary and testGenerateContent_success_thinking_functionCall_thoughtSummaryAndSignature to validate thought summary and signature parsing.
    • Added streaming unit tests testGenerateContentStream_successWithThoughtSummary and testGenerateContentStream_success_thinking_functionCall_thoughtSummary_signature.
  • FirebaseAI/Tests/Unit/GenerativeModelVertexAITests.swift
    • Added unit tests testGenerateContent_success_thinking_thoughtSummary and testGenerateContentStream_successWithThinking_thoughtSummary for Vertex AI.
Activity
  • The PR author, andrewheard, initially marked the PR as 'WIP - needs tests', but has since added comprehensive tests as seen in the patch.
  • A Danger bot warning was posted regarding a missing changelog entry, which can be silenced by adding #no-changelog to the PR description.
  • Reviewer paulb777 inquired about the exclusion of 'thoughts' from the history. andrewheard clarified that while an internal chat suggested exclusion, they decided to align with the Python SDK's approach of including them, leading to a change in the aggregation logic.
  • paulb777 also questioned the use of an optional Bool for isThought. andrewheard provided a detailed explanation, citing potential backend validation changes and consistency with other SDKs (Python, TypeScript/JavaScript, Go, Java) where includeThoughts is also optional. They concluded in an offline discussion that the optional Bool is acceptable.

@andrewheard andrewheard marked this pull request as ready for review August 18, 2025 22:10
@andrewheard
Copy link
Contributor Author

/gemini review

gemini-code-assist[bot]
gemini-code-assist bot reviewed Aug 18, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for thought summaries from thinking-compatible models. The changes are well-implemented, introducing a new thoughtSummary property on GenerateContentResponse and updating the ThinkingConfig and internal data structures like ModelContent and Part to handle thought-related metadata. The refactoring of InternalPart to be a standalone struct with a nested OneOfData enum is a significant improvement in code structure and clarity. The addition of comprehensive unit and integration tests ensures the new functionality is well-covered. I have a few minor suggestions for refactoring to improve code conciseness and a fix for a test assertion.

paulb777
paulb777 reviewed Aug 18, 2025
View reviewed changes
@andrewheard andrewheard merged commit e176f01 into main Aug 18, 2025
47 checks passed
@andrewheard andrewheard deleted the ah/firebaseai-thoughts branch August 18, 2025 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@paulb777 paulb777 paulb777 approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
api: firebaseai
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@andrewheard @paulb777